refactor(tensor-pool): move behind a generic extension seam, opt-in and outside the 1.0 guarantees - #3152

Moves the pinned/CUDA tensor-pool transport (#2168, #2386, #2619) out of dora's core and behind a generic extension seam. It stays in the tree and stays usable — as an opt-in feature, explicitly outside the 1.0 compatibility guarantees.

Three commits, each independently reviewable:

1. Remove the transport from dora's core — 3,048 lines from the Python binding, 973 from the daemon, five wire-protocol variants.
2. Add a generic extension channel — the one thing an out-of-core transport genuinely needs from the daemon.
3. Reinstate the transport at libraries/extensions/tensor-pool, on that channel, behind --features tensor-pool.

Why

#1872 — the design issue for this feature — explicitly declined to commit to an architecture:

This issue documents the gap and invites proposals. It does NOT commit to an architecture. dora has historically moved AWAY from custom shared-memory infrastructure (see #1745…). Any new transport has to clear a high bar.

Priority: not currently scheduled. Filed for visibility.

It was filed 2026-05-19, the same day #1623 (+2,717 lines) was closed for being "7× larger than the feature needed", and listed five questions a proposal had to answer before writing code. #2168 landed the same architecture 24 days later, merged 5 days after opening, answering none of them. Question 3 — "who owns pinned memory, and when is it freed? what if the producer crashes mid-write?" — is the direct ancestor of #2881, #2935 and #3015.

The problem was never the feature. It was that the feature had grown into dora's core: ~3,000 lines inside apis/python/node/src/lib.rs with 64 unsafe sites (~40 hand-parsing a 256-byte binary header at hardcoded offsets, out of memory another process writes), 950 lines of daemon lifecycle logic, pool-specific variants frozen into the wire protocol — and no CI able to exercise any of its GPU paths.

So: keep the feature, remove it from the core, and make the boundary explicit.

1. The core is clean again

grep -ri 'cuda\|doradma\|pinned\|seqlock' over dora's core now returns nothing.

2. The seam

A transport outside the core still needs one thing only the daemon can provide: reclamation after a crash. A node that dies cannot withdraw the descriptor it published, so its readers keep mappings to memory nobody owns. That is #2881, and send_output cannot fix it.

So dora gained a dataflow-scoped table of opaque byte values whose lifetime the daemon brokers — and nothing else:

node.extension_store(namespace, key, value)          # bytes in
node.extension_load(namespace, key, remove=False)    # bytes out, or None
node.extension_drop(namespace, key)                  # withdraw + notify
node.drain_dropped_extension_keys(namespace)         # what went away

Same four on DoraNode for Rust nodes. Guarantees and limits: docs/extensions.md.

Deliberately generic — the variants are ExtensionStore / ExtensionLoad / ExtensionDrop, and adding a second extension needs no change to dora at all. A pool-shaped accessor would have frozen this transport's architecture into the framework, which is exactly what #1872 declined to do.

17 tests: ownership (a foreign node cannot hijack a key), namespace and dataflow scoping, the reclamation asymmetry (an owner's exit reclaims, a reader's must not), idempotent drop, per-dataflow cap.

3. The transport, opt-in

Lives at libraries/extensions/tensor-pool — beside ros2-bridge, which has the same python/ subcrate shape.

maturin develop -m apis/python/node/Cargo.toml --features tensor-pool
cargo build -p dora-daemon --features tensor-pool   # orphaned-segment reclamation

Both flags are off by default and independent. The default build neither compiles nor exposes any of it: process_pending_tensor_pool_frees becomes an empty no-op, and default cargo check on the daemon and Python crate is warning-clean.

register_tensor_pool and friends are still methods on Node, #[cfg(feature)]-gated, delegating to a Pool context struct (PyO3 cannot let another crate add #[pymethods]).

Renamed from memory-pool. dora already has an unrelated shared_memory_pool_size descriptor key (and DORA_NODE_SHM_POOL_SIZE) for the Zenoh SHM buffer pool, and "memory pool" for both was a persistent source of confusion. tensor-pool says what this one actually pools and stays accurate on the CPU path, which gpu- would not — the CPU path works without CUDA and is the only one with CI coverage.

This renames the four Python methods (register_memory_pool → register_tensor_pool, and likewise for write/read/free), the crates, the feature flags and the example env keys. That is a user-visible break for anyone on the current methods — permissible precisely because this feature sits outside the 1.0 guarantees, and better done now than after more users arrive.

Its six former daemon calls now go through the seam:

The descriptor is JSON that dora never parses. That boundary is what keeps the unsafe pointer arithmetic, the seqlock and the embedded libcudart bindings on the extension's side.

Guarantee status is stated where it gets read: the README leads with it, and it is repeated in the crate description, the module header, every #[pymethods] docstring and the .pyi stubs. Named open defects: #3015, #2935, #2890. (#2881 is fixed.)

Three latent bugs surfaced

Returning the crate to the workspace put it under -D warnings for the first time, which caught: a deprecated downcast_into, a dead initializer in the device-to-host copy path, and seqlock_begin_write with no callers at all — every write path uses begin_if_even. All fixed; the dead function is deleted.

Known gap

The smoke tests are not wired into any automated suite. They need a feature-built wheel plus torch, so as a cargo test target they broke the build. They ship as smoke-tests.rs.example with instructions instead. The extension therefore has 53 unit tests but no in-tree end-to-end coverage — a real regression against the old nightly memory-pool-smoke job, and worth tracking separately.

Validation

cargo fmt --all -- --check, cargo clippy --all -- -D warnings, cargo check --examples — clean, on both the default and --features tensor-pool builds.

cargo test --all: two failures, dora-ros2-bridge --test rmw_zenoh_pubsub::{transient_local_delivers_history_to_late_joiner, two_sessions_preserve_order_payload_and_metadata}, both Elapsed(()) on Zenoh peer discovery. They reproduce identically on an unmodified checkout — the dev container has no multicast — and this branch touches zero lines in that crate. The apis/c++ build-script failure in that environment is likewise pre-existing on main.